Table of Contents:

• 1. List Comprehensions
  • 1.1. Nim - Comprehending a list with a for loop
  • 1.2. Python's List Comprehension structure
  • 1.3. Python - Comprehending a list with a for loop
  • 1.4. Nim - Using sugar.collect
  • 1.5. Python - List Comprehension
• 2. Python's list comprehensions vs Nim's collect
  • 2.1. Python's List Comprehension Cons
• 3. Nim's sugar collect Pros
  • 3.1. Long one liners are bad
  • 3.2. Sugar collect also works on Sets and Tables
  • 3.3. collect with Sets
  • 3.4. collect with hashSets
  • 3.5. collect with Tables
  • 3.6. sugar.collect as a one-liner
• 4. Performance Benchmarks
  • 4.1. Nim's 3 variants
  • 4.2. Python's 3 variants
  • 4.3. Why doesn't Nim have proper list comprehensions ?

INTRO - GREETING

• Title: List Comprehensions with sugar.collect and comparison against Python's List Comprehensions
• Alt Title: List Comprehensions with sugar.collect vs Python's List Comprehensions

INTRO - FOREWORDS

(What is the purpose of this video ?)

• List comprehensions with sugar.collect and comparison against Python's List Comprehensions:
  • In this video we will go over sugar.collect, it's pros and cons, and compare it against Python's list comprehensions with some benchmarks.

The code for this video and it's script/documentation styled with nimib, is in the link in the description as a form of written tutorial.

1. List Comprehensions

sugar.collect is a macro, a macro used for list comprehensions. A list comprehension is basically making a list from an existing list based on some variable parameters. This can be done with either a for loop or sugar.collect, list comprehensions in Python.

  1.1. Nim - Comprehending a list with a for loop

In the following example, we make a new list/sequence based on the fruit sequence, by iterating over it with a for loop using the pairs iterator, and returning only the elements of the list starting with the letter "P".

let fruit = @["Apple", "Orange", "Banana", "Peach", "Pear", "Pineapple"]

var pFruit: seq[string]

for i, f in fruit.pairs:
  if $f[0] == "P":
    pFruit.add f

echo pFruit

@["Peach", "Pear", "Pineapple"]

  1.2. Python's List Comprehension structure

Before we go on for the Python's version of the above example, here is the structure of a Python's list comprehension, with explanation.

  list = [returned for element in container if condition]

• returned - the element/value that will be returned/put in a new list.
• element - element of "a" container.
• container - dict, list, set, tuple(in Nim, it's a data structure).
• condition - optional -> to perform operations for the "returned".

  1.3. Python - Comprehending a list with a for loop

  fruit = ["Apple", "Orange", "Banana", "Peach", "Pear", "Pineapple"]
  pfruit = []

  for f in fruit:
    if "P" in f:
      pfruit.append(f)

  print(pfruit)

  ['Peach', 'Pear', 'Pineapple']

  1.4. Nim - Using sugar.collect

The exact same code as the one with the for loop, except now it uses the sugar module's collect macro.

let pFruit2 = collect(newSeq):
  for i, f in fruit.pairs:
    if $f[0] == "P":
      f

echo pFruit
echo pFruit2

@["Peach", "Pear", "Pineapple"]
@["Peach", "Pear", "Pineapple"]

  1.5. Python - List Comprehension

  fruit2 = ["Apple", "Orange", "Banana", "Peach", "Pear", "Pineapple"]

  pfruit2 = [f for f in fruit2 if "P" in f]

  print(pfruit)
  print(pfruit2)

  ['Peach', 'Pear', 'Pineapple']
  ['Peach', 'Pear', 'Pineapple']

2. Python's list comprehensions vs Nim's collect

Python's List comprehensions are a more compact way to create new lists, in situations where map() and filter(), and or nested loops would be used. They are great for simple tasks.

Here are a few examples:

  print([i for i in range(10)])

  toTen = []
  for i in range(10):
    toTen.append(i)

  print(toTen)

  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

  print([i for i in range(20) if i%2 == 0])

  evenToTwenty = []
  for i in range(20):
    if i%2 == 0:
      evenToTwenty.append(i)

  print(evenToTwenty)

  [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

  nums = [1, 2, 3, 4]
  fruit = ["Apples", "Peaches", "Pears", "Bananas"]
  print([(i, f) for i in nums for f in fruit])

  fourTimesFour = []
  for i in nums:
    for f in fruit:
      fourTimesFour.append((i, f)) #Double (()) - because tuple as argument - python demands () for funcs

  print(fourTimesFour)

  [(1, 'Apples'), (1, 'Peaches'), (1, 'Pears'), (1, 'Bananas'),
  (2, 'Apples'), (2, 'Peaches'), (2, 'Pears'), (2, 'Bananas'),
  (3, 'Apples'), (3, 'Peaches'), (3, 'Pears'), (3, 'Bananas'),
  (4, 'Apples'), (4, 'Peaches'), (4, 'Pears'), (4, 'Bananas')]

  print([(i, f) for i in nums for f in fruit if f[0] == "P"])

  fruitP = []
  for i in nums:
    for f in fruit:
      if f[0] == "P":
        fruitP.append((i, f))

  print(fruitP)

  [(1, 'Peaches'), (1, 'Pears'),
  (2, 'Peaches'), (2, 'Pears'),
  (3, 'Peaches'), (3, 'Pears'),
  (4, 'Peaches'), (4, 'Pears')]

  print([(i, f) for i in nums for f in fruit if f[0] == "P" if i%2 == 1])

  fruitPodd = []
  for i in nums:
    for f in fruit:
      if f[0] == "P" and i%2 == 1:
        fruitPodd.append((i, f))

  print(fruitPodd)

  [(1, 'Peaches'), (1, 'Pears'), (3, 'Peaches'), (3, 'Pears')]

  print([i for i in zip(nums, fruit) if i[0] %2== 0])

  fruitEvenZip = []

  for i in zip(nums, fruit):
    if i[0] %2 == 0:
      fruitEvenZip.append(i)

  print(fruitEvenZip)

  [(2, 'Peaches'), (4, 'Bananas')]

  2.1. Python's List Comprehension Cons

Python's List Comprehension are useful for small and simple lists, but become extremely hard to understand the more complex they become, unlike just using a for loop, which is yes more verbose, but also easier to understand and read, especially if you haven't viewed the code with with a list comprehension for a while.

Here is a good example of a bad one liner:

  listLLL = [[[1,2,3],[4,5,6],[7,8,9]]]
  flatten_keep_evens = [item for listLL in listLLL for listL in listLL for item in listL if item % 2 == 0]

  print(flatten_keep_evens)

  [2, 4, 6, 8]

The above good example of a bad one liner, can be greatly improved by indentation(like a for loop) like this:

  doubleNestedLLL = [[[1,2,3], [4,5,6], [7,8,9]]]
  flatenedEvens = [
      element
      for nestedLL in doubleNestedLLL
      for elementsL in nestedLL
      for element in elementsL
      if not element % 2
  ]

  print(flatenedEvens)

  [2, 4, 6, 8]

3. Nim's sugar collect Pros

  3.1. Long one liners are bad

Nim does not do list comprehensions as one liners like Python does, they work very much the same as a for loop, but reduce a bit of typing, by not having to declare/initialize a sequence before hand, but at the moment of usage. Then we also don't need to call .add + elementWeWant, but just the elementWeWant.

  3.2. Sugar collect also works on Sets and Tables

Nim's sugar's collect also works on hashSets and Tables, unlike Python. Nim's collect macro does not have the drawbacks of Python's list comprehensions, while being much more powerful, because collect also works on other types. These types being, Sets, HashSets and Tables.

  3.3. collect with Sets

var normalSet = {1, 2, 3, 4, 5, 6}

var evenNumbers = collect:
  for num in normalSet:
    if num mod 2 == 0:
      num

echo evenNumbers

@[2, 4, 6]

  3.4. collect with hashSets

import std/sets

let keyValueSet = [("Key 1", 1), ("Key 2", 2), ("Key 3", 3), ("Key 4", 4), ("Key 5", 5)].toOrderedSet

Normal version:

var oddKeys: HashSet[(string, int)]

for tup in keyValueSet:
  if tup[1] mod 2 != 0:
    oddKeys.incl tup

echo oddKeys

{("Key 1", 1), ("Key 3", 3), ("Key 5", 5)}

collect version:

var oddKeysCollect = collect:
  for tup in keyValueSet:
    if tup[1] mod 2 != 0:
      tup

echo oddKeysCollect

@[("Key 1", 1), ("Key 3", 3), ("Key 5", 5)]

  3.5. collect with Tables

import std/tables

let characterClassEquipment = {"Paladin": @["Maul", "Breastplate"], 
                              "Wizard": @["Staff", "Robe"], "Druid": @["Staff"],
                              "Fighter": @["Sword", "Shield", "Plate Armour"]}.toOrderedTable

Normal version:

var staffWielders = initOrderedTable[string, seq[string]]() #Do NOT forget `()` at the end

for class, items in characterClassEquipment:
  for item in items:
    if item == "Staff":
      staffWielders[class] = items #`table.add key, value` is now deprecated(marked for future removal)

echo staffWielders

{"Wizard": @["Staff", "Robe"], "Druid": @["Staff"]}

collect version:

var staffWieldersCollect = collect:
  for class, items in characterClassEquipment:
    for item in items:
      if item == "Staff":
        {class: items} #Kind of bizzare way of returning

echo staffWieldersCollect

{"Wizard": @["Staff", "Robe"], "Druid": @["Staff"]}

  3.6. sugar.collect as a one-liner

You can also write sugar.collect as a one-liner like this:

let evenNumbs = collect(for i in [0, 1, 2, 3, 4]: (if i mod 2 == 0: i))
echo evenNumbs

@[0, 2, 4]

Credits to: "razorgamedev" for pointing that out to me on the reddit post.

4. Performance Benchmarks

Now i will show you performance benchmarks of 3 variants of making a new list, in both Python and Nim. This is NOT a benchmark of the languages. This is only meant to see the difference of the 3 variants.

My relevant computer specs used for this benchmark:

• CPU: Ryzen 2700x
• RAM: G.Skill Ripjaws V 16GB (2x8GB) DDR4 F4-3200C16D-16GVKB

The 3 variants are as follow:

• add and append
• index - []
• collect and list comprehension

All 3 variants of the 2 versions of Nim and Python, create a new list by filling a pre-initialized to size 10000 sequence/array, with arbitrary data, that is then benchmarked 10000 times, in order to account for slight variations that occur. The resulting time elapsed is then divided back by 10000, to get the mostly correct/aproximate time elapsed for our variations.

This is eased for Nim by the usage of a custom template, while Python has an official module specifically for this purpose.

  4.1. Nim's 3 variants

import std/times, std/os, std/strutils, std/sequtils, std/sugar

template benchmark(benchmarkName: string, timesToRun: int, code: untyped) =
  block:
    let t0 = cpuTime()

    for run in 0..timesToRun:
      code

    let elapsed = cpuTime() - t0
    let elapsedProcessed = elapsed / timesToRun
    let elapsedProcessedStr = elapsedProcessed.formatFloat(format = ffDecimal, precision = 12)
    echo "CPU Time [", benchmarkName, "] x ", timesToRun, " = ", elapsedProcessedStr, "s"

benchmark "add    ", 10000:
  let num = 10000
  var list: seq[num.typeof]

  for j in 0..num:
    list.add j

benchmark "index  ", 10000:
  let num = 10000

  #Initializing a seq of 10000 ups the time by about 3x
  var list = newSeqWith(num, 0) #size, default value

  for index, ele in list.pairs:
    list[index] = index

benchmark "collect", 10000:
  let num = 10000

  var list = collect:
    for i in 0..num:
      i

CPU Time [add    ] x 10000 = 0.000166400000s
CPU Time [index  ] x 10000 = 0.000138800000s
CPU Time [collect] x 10000 = 0.000165400000s

  4.2. Python's 3 variants

  from timeit import timeit

  def fAppend():
      num = 10000
      li = []
      for i in range(num):
          li.append(i)

  def fIndex():
      num = 10000
      li = [0] * num
      for i in range(num):
          li[i] = i

  def fLComprehension():
      num = 10000
      li = [i for i in range(num)]

  def main(x):
      print(timeit(stmt = fAppend, number = x) / x)
      print(timeit(stmt = fIndex, number = x) / x)
      print(timeit(stmt = fLComprehension, number = x) / x)

  main(10000) 

  CPU Time [append       ] x 10000 = 0.00065512317
  CPU Time [index        ] x 10000 = 0.0003905595900000001
  CPU Time [comprehension] x 10000 = 0.0002650251300000001

(You may run the above benchmarks yourself to double check the data. But the results will vary by your CPU and RAM speeds.)

  4.3. Why doesn't Nim have proper list comprehensions ?

Nim doesn't have list comprehensions as part of the language, because what can be done as a macro, should be done as a macro.

This is a part of Nim's philosophy of keeping the language as small as possible, while implementing what is not required or needed as a macro in a usually separate module.

Outro - Afterwords

Okay, that's it for this video, thanks for watching like, share and subscribe, aswell as click the bell icon if you liked it and want more, you can also support me on Patreon if you so wish. If you had any problems with any part of the video, let me know in the comment section, the code of this video, script and documentation, are in the link in the description, as a form of written tutorial.

Thanks to my past and current Patrons

Past Patrons

• Goose_Egg: From April 4th 2021 to May 10th 2022
• Davide Galilei(1x month)

Current Patrons

• jaap groot (from October 2023)
• Dimitri Lesnoff (from October 2023)

Compiler Information

• Version used: E.G. 2.0.2
• Compiler settings used: none, ORC is now the default memory management option(mm:orc)

My and General Links

• Patreon
• Visual Studio Code Shortcuts

Links to this video's subject:

• Sugar module's collect macro
• Python's List Comprehensions
• Python's map() function
• Python's filter() function
• Python's benchmarking timeit module's timeit() function
• Nim's formatFloat func
• Nim's cpuTime from the Times module